Measurement device and measurement method

ABSTRACT

According to an embodiment, a measurement device includes a processing circuitry. A plurality of images are captured in time series by an image capturing unit installed in a moving object. The processing circuitry identifies a region in which other moving object moving in surroundings of the moving object is present for each of the images, based on position and direction information of the moving object, and moving object information of the other moving object. The processing circuitry estimates position and posture of the image capturing unit based on the images. The processing circuitry searches for sets of corresponding points among non-moving object regions in the respective images. The processing circuitry performs 3D measurement based on the position and posture of the image capturing unit and the sets of the corresponding points.

CROSS-REFERENCE TO RELATED APPLICATION

This application is based upon and claims the benefit of priority fromJapanese Patent Application No. 2015-183874, filed on Sep. 17, 2015; theentire contents of which are incorporated herein by reference.

FIELD

An embodiment described herein relates generally to a measurement deviceand a measurement method.

When other moving object is present in captured images, the conventionaltechniques, however, cause accuracy of three-dimensional measurement todeteriorate because the position of the other moving object differsamong the images as a result of the movement of the other moving object.

BACKGROUND

Techniques have been known that implement three-dimensional measurementusing a plurality of images captured in time series by a camerainstalled in a moving object such as a vehicle that is running and amovement amount of the moving object.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram illustrating exemplary structures of ameasurement device and a moving object in an embodiment;

FIG. 2 is a schematic diagram illustrating an example of the movingobject in the embodiment;

FIG. 3 is a schematic diagram illustrating an example of modelinformation in the etch embodiment;

FIG. 4 is a schematic diagram illustrating an exemplary image in theembodiment;

FIG. 5 is an explanatory view illustrating an example of identifyingmoving object regions in the embodiment;

FIG. 6 is an explanatory view illustrating other example of identifyingthe moving object regions in the embodiment;

FIG. 7 is an explanatory view illustrating still other example ofidentifying the moving object regions in the embodiment;

FIG. 8 is an explanatory view illustrating an exemplary technique fortracking feature points in the embodiment;

FIG. 9 is an explanatory view illustrating the exemplary technique fortracking the feature points in the embodiment;

FIG. 10 is an explanatory view illustrating an exemplary technique forsearching for corresponding points in the embodiment;

FIG. 11 is an explanatory view illustrating an exemplary technique forextracting three-dimensional points in the embodiment;

FIG. 12 is an explanatory view illustrating an exemplary technique fordividing a space in the embodiment;

FIG. 13 is an explanatory view illustrating an exemplary technique forselecting a representative point in the embodiment;

FIG. 14 is an explanatory view illustrating an exemplary technique forestimating a movement plane in the embodiment;

FIG. 15 is an explanatory view illustrating an exemplary technique fordetecting an obstacle in the embodiment;

FIG. 16 is a flowchart illustrating exemplary processing in theembodiment;

FIG. 17 is an explanatory view illustrating a comparative example of theembodiment;

FIG. 16 is an explanatory view illustrating an example of the advantagesof the embodiment; and

FIG. 19 is a schematic diagram illustrating an exemplary hardwarestructure of the measurement device in the embodiment.

DETAILED DESCRIPTION

According to an embodiment, a measurement device includes a processingcircuitry. The processing circuitry acquires a plurality of imagescaptured in time series by an image capturing unit installed in a movingobject. The processing circuitry acquires first position informationthat indicates a position of the moving object and first directioninformation that indicates a direction of the moving object. Theprocessing circuitry acquires moving object information that includessecond position information indicating a position of other moving objectmoving in surroundings of the moving object. The processing circuitryidentifies a moving object region in which the other moving object ispresent for each of the images, based on the first position information,the first direction information, and the moving object information. Theprocessing circuitry estimates a position and a posture of the imagecapturing unit based on the images. The processing circuitry searchesfor a plurality of sets of corresponding points among non-moving objectregions other than the moving object regions in the respective images.The processing circuitry performs three-dimensional measurement based onthe position and the posture of the image capturing unit and the sets ofthe corresponding points.

The following describes an embodiment in detail with reference to theaccompanying drawings.

FIG. 1 is a schematic diagram illustrating an exemplary structure of ameasurement device 10 according to the embodiment and an exemplarystructure of a moving object 1 provided with the measurement device 10.FIG. 2 is a schematic diagram illustrating an example of the movingobject 1 in the embodiment. As illustrated in FIG. 1, the measurementdevice 10 is installed in the moving object 1 that includes an imagecapturing unit 5 and an azimuth sensor 6. The measurement device 10includes a first acquirer 11, a second acquirer 13, a third acquirer 15,an identifier 17, a first estimator 18, a searcher 19, a measurer 21, asecond estimator 23, a detector 25, and an output unit 27.

In the embodiment, the moving object 1 is a vehicle such as a motorvehicle that moves on a road surface serving as a movement plane, forexample. The moving object 1 is, however, not limited to this example.The moving object 1 may be any object that can move on the movementplane. For example, the moving object 1 may be a ship that moves on awater surface serving as the movement plane or a robot that moves on afloor surface serving as the movement plane.

The image capturing unit 5 can be achieved by an image sensor or acamera, for example. The image capturing unit 5 captures images of thesurroundings (e.g., in a traveling direction of the moving object 1) ofthe moving object 1 in time series and outputs a plurality of capturedimages to the measurement device 10.

The azimuth sensor 6 detects a direction of the moving object 1 andoutputs, to the measurement device 10, first position information thatindicates the detected direction of the moving object. 1

The first acquirer 11, the second acquirer 13, the third acquirer 15,the identifier 17, the first estimator 18, the searcher 19, the measurer21, the second estimator 23, the detector 25, and the output unit 27 maybe achieved by a processing unit such as a central processing unit (CPU)executing a program, i.e., by software, by hardware such as anintegrated circuit (IC), or by both of the software and the hardware.The measurement device 10 may be achieved by a chip (integrated circuit)or a typical computer.

The first acquirer 11 acquires, from the image capturing unit 5, themultiple images captured in time series.

The second acquirer 13 sequentially acquires the first positioninformation that indicates the position of the moving object 1 and firstdirection information that indicates the direction of the moving object1. The second acquirer 13 acquires the first position information fromthe global positioning system (GPS) satellites and the first directioninformation from the azimuth sensor 6, for example. The acquisitionmanner of the first position information and the first directioninformation is not limited to the example.

The third acquirer 15 acquires information about other moving object(hereinafter referred to as moving object information). The movingobject information includes second position information indicating theposition of other moving object moving in the surroundings of the movingobject 1. The moving object information may further include at least oneof speed information that indicates a speed of the other moving object,model information that indicates a model obtained by abstracting a shapeof the other moving object, texture information that indicates at leastone of a color and a pattern of the other moving object, and seconddirection information that indicates a direction of the other movingobject. The model information may indicate at least one of the width,the height, and the depth (the length) of the other moving object asillustrated in FIG. 3. In this case, the model information indicates amodel obtained by abstracting the shape of the other moving object to arectangular parallelepiped. The model information may indicate a modelof a cube obtained by multiplying any length of the width, the height,and the depth of the other moving object by a constant, or a model of arectangular parallelepiped obtained by estimating a length of the otherside from any length of the width, the height, and the depth of theother moving object using a general aspect ratio of a vehicle.

The third acquirer 15 sequentially acquires the moving objectinformation from other moving object by performing inter-vehiclecommunication (e.g., wireless communication according to IEEE 802.21p)with the other moving object running in the surroundings of the movingobject 1, for example. In this case, the other moving object acquiresthe second position information from the GPS satellites, puts theacquired second position information in the moving object information,and transmits the moving object information to the moving object 1 (themeasurement device 10), for example. The other moving object may detecta speed thereof, produce the speed information, and put the producedspeed information in the moving object information. When preliminarilyholding the model information and the texture information thereof, theother moving object may put them in the moving object information. Theother moving object may acquire the second direction information from anazimuth sensor included therein, and put the acquired second directioninformation in the moving object information.

The third acquirer 15 may sequentially acquire the moving objectinformation from the other moving object moving in the surroundings ofthe moving object 1 from a monitoring device by performing road-vehiclecommunication (e.g., wireless communication according to IEEE 802.21p)with the monitoring device present on a road shoulder in thesurroundings of the moving object 1. The monitoring device is amonitoring camera, for example. The monitoring device is, however, notlimited to the example. In this case, the monitoring device captures animage of the other moving object moving in the surroundings of themoving object 1, calculates the second position information using thecaptured image, puts the calculated second position information in themoving object information, and transmits the moving object informationto the moving object 1 (the measurement device 10). The monitoringdevice may use the position information about the monitoring deviceacquired from the GPS satellites for calculating the second positioninformation. The monitoring device may calculate the model information,the texture information, and the second direction information using thecaptured image, and put the calculated information in the moving objectinformation.

The identifier 17 identifies a moving object region in which the othermoving object is present for each of the multiple images acquired by thefirst acquirer 11 on the basis of the first position information and thefirst direction information that are acquired by the second acquirer 13and the moving object information acquired by the third acquirer 15.

Specifically, the identifier 17 determines that the other moving objectis moving when a speed indicated by the speed information included inthe moving object information is equal to or larger than a firstthreshold. When no speed information is included in the moving objectinformation, the identifier 17 may calculate a speed to be included inthe moving object information from a difference between the positionindicated by the second position information included in the movingobject information at this time and the position indicated by the secondposition information included in the moving object information atprevious time, and an acquisition interval of the moving objectinformation.

When determining that the other moving object is moving, the identifier17 identifies the position of the other moving object on an imageacquired by the first acquirer 11 on the basis of the first positioninformation and the first direction information that are acquired by thesecond acquirer 13 and the second position information included in themoving object information acquired by the third acquirer 15. Theidentifier 17 can identify a positional relation between the movingobject 1 and the other moving object on an image captured by the imagecapturing unit 5 of the moving object 1 because the position and thedirection of the moving object 1 and the position of the other movingobject are known, thereby making it possible to identify the position ofthe other moving object on the image. The first position information,the first direction information, the moving object information, and theimage are captured at substantially the same time.

The identifier 17 further identifies the moving object region in whichthe other moving object is present on the image on the basis of theidentified position of the other moving object on the image.

Specifically, the identifier 17 identifies, as the moving object region,a region having a predetermined size based on the identified position ofthe other moving object. For example, the identifier 17 identifies, asthe moving object region, a region that includes the identified positionof the other moving object as the center and the predetermined number ofpixels. The moving object region may have any shape such as rectangularor circular.

The identifier 17 may identify, as the moving object region, a regionthat has a size according to a distance between the moving object 1 andthe other moving object based on the identified position of the othermoving object. For example, when identifying the positions (centerpositions) of moving bodies 31 and 32 serving as the other moving bodiesas illustrated in FIG. 4, the identifier 17 may identify a moving objectregion 41 having the position of the moving object 31 as the centerthereof and a moving object region 42 having the position of the movingobject 32 as the center thereof as illustrated in FIG. 5. In theexamples illustrated in FIGS. 4 and 5, the size (the number of pixels)of the moving object region 42 is smaller than that of the moving objectregion 41 because the distance from the moving object 1 to the movingobject 32 is larger (farther) than that from the moving object 1 to themoving object 31. The distance between the moving object 1 and the othermoving object is obtained on the basis of the first position informationand the corresponding second position information.

The identifier 17 may identify, as the moving object region, a regionaccording to the model of the other moving object based on theidentified position of the other moving object. The identifier 17 mayidentify, as the moving object region, a region according to thedirection of the other moving object. The identifier 17 may identify, asthe moving object region, a region according to the distance between theother moving object and the moving object 1. For example, whenidentifying the positions (center positions) of the moving bodies 31 and32 serving as the other moving bodies as illustrated in FIG. 4, theidentifier 17 may identify a moving object region 51 having the positionof the moving object 31 as the center thereof and a moving object region52 having the position of the moving object 32 as the center thereof asillustrated in FIG. 6.

In the example illustrated in FIG. 6, the moving object region 51 isidentified according to the model and the direction of the moving object31 and the distance between the moving object 31 and the moving object1. Specifically, the moving object region 51 is identified in such amanner that the model and the direction of the moving object 31 are setin a three-dimensional space and the model is projected such that themoving object 31 is positioned at the center of the moving object region51. Likewise, the moving object region 52 is identified according to themodel and the direction of the moving object 32 and the distance betweenthe moving object 32 and the moving object 1. Specifically, the movingobject region 52 is identified in such a manner that the model and thedirection of the moving object 32 are set in a three-dimensional spaceand the model, is projected such that the moving object 32 is positionedat the center of the moving object region 52. The model of the othermoving object can be obtained from the model information while thedirection of the other moving object can be obtained from the seconddirection information.

When the region according to the model of the moving object 31 and thedistance between the moving object 31 and the moving object 1 isidentified as the moving object region, the projection may be performedsuch that a length of the depth of the model projected on the image is alength of a rectangle in a lateral direction while a length of theheight of the model projected on the image is a length of the rectanglein the height direction.

The identifier 17 may identify, as the moving object region, a regionaccording to the texture information about the other moving object basedon the identified position of the other moving object. For example, whenidentifying the positions (center positions) of the moving bodies 31 and32 serving as the other moving bodies as illustrated in FIG. 4, theidentifier 17 may identify a moving object region 61 of the movingobject 31 and a moving object region 62 of the moving object 32 asillustrated in FIG. 7.

In the example illustrated in FIG. 7, the moving object region 61 isidentified according to the texture information about the moving object31. Specifically, the moving object region 61 has a color the same as ora similar to that indicated by the texture information around theposition of the moving object 31. Likewise, the moving object region 62is identified according to the texture information about the movingobject 32. Specifically, the moving object region 62 has a color thesame as or a similar to that indicated by the texture information aroundthe position of the moving object 32.

The first estimator 18 estimates the position and the posture of theimage capturing unit 5 on the basis of the multiple images acquired bythe first acquirer 11. Specifically, the first estimator 18 extractsfeature points from the respective images, tracks the feature pointsamong the images, and estimates the position and the posture of theimage capturing unit 5. In the embodiment, the first estimator 18extracts the feature points from non-moving object regions other thanthe moving object regions in the respective images, tracks the featurepoints among the images, and estimates the position and the posture ofthe image capturing unit 5.

FIG. 8 illustrates the image captured at time t−1. FIG. 9 illustratesthe image captured at time t. In the examples illustrated in FIGS. 8 and9, the first estimator 18 extracts the feature points from thenon-moving object region other than a moving object region 74 asillustrated in FIG. 8 and tracks the extracted feature points on theimage illustrated in FIG. 9. In the examples, the first estimator 18extracts feature points 71, 72, and 73 from the image illustrated inFIG. 8, tracks the respective feature points 71, 72, and 73 on the imageillustrated in FIG. 9, and extracts feature points 81, 82, and 83,respectively, from the image illustrated in FIG. 9. The first estimator18 obtains a correspondence in time series between the correspondingfeature points on the corresponding images for each feature point. Thefirst estimator 18 obtains the position and the posture (currentposition and posture) of the image capturing unit 5 relative to theposition and the posture of the image capturing unit 5 at previous timeon the basis of the obtained correspondences of the respective featurepoints in time series (posture rotation R and parallel movement T) byepipolar geometry.

A point having a large luminance difference in an image (e.g., a pointhaving a large luminance difference both in lateral and longitudinaldirections) can be chosen as the feature point. The feature points canbe extracted by a known extraction technique such as Harris cornerdetector.

In the embodiment, a technique that associates the feature points witheach other is described as an example of tracking the feature points.The tracking the feature points includes a technique in which a regionsurrounding a certain pixel is considered as the feature point, and thepixels are associated with each other between the regions or the pixelsare associated with each other using the region and the feature point.

The searcher 19 searches for a plurality of sets of the correspondingpoints among the non-moving object regions in the respective imagesacquired by the first acquirer 11. Specifically, the searcher 19arranges search points in the non-moving object region on the imagecaptured at previous time, and searches for corresponding points eachcorrespond to one of the search points on the image captured at thistime to determine the sets of the corresponding points.

The search points may be arranged in the non-moving object region on theimage captured at previous time such that the search points are arrangedentirely on the periphery of the non-moving object region or evenly andentirely in the non-moving object region. The searcher 19 can identify arange in which the search points are capable of being observed on theimage captured at this time from the position and the posture, which areestimated by the first estimator 18, of the image capturing unit 5,thereby searching the identified range and searching for thecorresponding points or the search points. Specifically, the searcher 19searches for the corresponding points corresponding to the search pointson epipolar lines (indicated with the arrows in FIG. 10), on which thesearch points can be observed as illustrated in FIG. 10.

In the embodiment, the search points are arranged on the image capturedat previous time and the corresponding points corresponding to thesearch points are searched for on the image captured at this time. Thesearch points may be arranged on the image captured at this time and thecorresponding points corresponding to the search points may be searchedfor on the image captured at previous time.

The measurer 21 performs three-dimensional measurement on the basis ofthe position and the posture, which are estimated by the first estimator18, of the image capturing unit 5 and the multiple sets of thecorresponding points searched for by the searcher 19 to obtainthree-dimensional points. Specifically, the measurer 21 performsthree-dimensional measurement based on a principle of triangulationusing the position and the posture, which are estimated by the firstestimator 18, of the image capturing unit 5 and the multiple sets of thecorresponding points searched for by the searcher 19. As a result, themeasurer 21 restores the depths of the respective corresponding points,thereby obtaining the three-dimensional points.

The second estimator 23 estimates a movement plane on which the movingobject 1 moves on the basis of the three-dimensional points obtained bythe measurer 21. For estimating the movement plane, a plane detectiontechnique using a random sample consensus (RANSAC), which is a knowntechnique, may be used for example. Specifically, a set of three pointshaving a height equal to or smaller than a second threshold is randomlyacquired at several times in a group of the three-dimensional pointsobtained by the measurer 21, and a plane that includes the largestnumber of three-dimensional points within a third threshold distancefrom the plane may be estimated as the movement plane out of the planeseach formed by the three points.

The movement plane can be estimated highly accurately using thefollowing technique.

The second estimator 23 sets a fourth threshold on the basis of a timeseries change in the posture, which is estimated by the first estimator18, of the image capturing unit 5 (the moving object 1). Specifically,the second estimator 23 sets the fourth threshold such that with anincrease in the time series change in the posture of the image capturingunit 5, the value of the fourth threshold is reduced. Specifically, thesecond estimator 23 sets, to the value of the fourth threshold, a valueobtained by applying a value indicating the time series change in theposture of the image capturing unit 5 to a monotonically decreasingfunction.

For example, let the monotonically decreasing function be y=−ax+b wherea and b are any desired variables. In this case, the second estimator 23calculates an absolute value of a difference between a value indicatingthe posture of the image capturing unit 5 of the moving object 1 at apresent time t and a value indicating the posture of the image capturingunit 5 of the moving object 1 at a calculation time t−P, which is thetime P hours before the present time t. The second estimator 23 sets, tothe fourth threshold, the value of y obtained by substituting thecalculated absolute value of the difference to x of the monotonicallydecreasing function.

The calculation time t−P is preferably the calculation time of theposture of the image capturing unit 5 at previous time, but is notlimited thereto. The calculation time is the calculation time of theposture of the image capturing unit 5 at or before the previous time.The absolute value of the difference is the sum of the absolute valuesof the differences in roll, pitch, and yaw, for example, but is notlimited thereto.

The second estimator 23 extracts a plurality of three-dimensional pointseach having a distance from the moving object 1 equal to or smaller thanthe set fourth threshold in the movement direction of the moving object1 out of the three-dimensional point group obtained by the measurer 21.

FIG. 11 is en explanatory view illustrating an exemplary technique forextracting the three-dimensional points in the embodiment, andillustrates a three-dimensional point group 101 obtained by the measurer21 on the yz plane. The movement direction of the moving object 1 is thez-axis direction (specifically, +z direction). In the exampleillustrated in FIG. 11, the second estimator 23 extracts a plurality ofthree-dimensional points 102 each having a z coordinate value equal toor smaller than the set fourth threshold T out of the three-dimensionalpoint group 101.

The second estimator 23 divides a space in which the extractedthree-dimensional points are positioned into a plurality of dividedspaces in the movement direction of the moving object 1.

FIG. 12 is an explanatory view illustrating an exemplary technique fordividing the space in the embodiment, and illustrates thethree-dimensional point group 101 and the three-dimensional points 102on the yz plane. In the example illustrated in FIG. 12, the secondestimator 23 identifies a minimum value L and a maximum value U out ofthe z coordinate values of the extracted three-dimensional points 102,and divides a space having an coordinate value equal to or larger than Land equal to or smaller than U equally into k (k≥2) in the z-axisdirection to obtain k block spaces. While FIG. 12 exemplifies a case ofU=T, it is sufficient if U satisfies a relation of U≤T.

The second estimator 23 selects, for each divided space, arepresentative point out of the three-dimensional points included in thedivided space. Specifically, the second estimator 23 selects, as therepresentative point, the three-dimensional point at the lowest positionin the vertical direction out of the three-dimensional points includedin the divided space.

FIG. 13 is an explanatory view illustrating an exemplary technique forselecting the representative point in the embodiment, and illustratesthe three-dimensional points 102 and the block spaces after the divisionon the yz plane. In the example illustrated in FIG. 13, the secondestimator 23 selects, as the representative point, the three-dimensionalpoint having a maximum y coordinate value in each block space. In afirst block space, the second estimator 23 selects three-dimensionalpoint 103-1 as the representative point. In a kth block space, thesecond estimator 23 selects three-dimensional point 103-k as therepresentative point.

The second estimator 23 estimates a plane approximated with the selectedrepresentative points as the movement plane on which the moving object 1moves.

FIG. 1.4 is an explanatory view illustrating an exemplary technique forestimating the movement plane in the embodiment, and illustrates theblock spaces after the division and k selected representative points 103on the yz plane. In the example illustrated in FIG. 14, the secondestimator 23 estimates a plane 104 (illustrated as the straight line onthe yz plane in FIG. 14) approximated with the k selected representativepoints 103 as the movement plane on which the moving object 1 moves.

The detector 25 detects an obstacle on the basis of thethree-dimensional point group obtained by the measurer 21 and themovement plane estimated by the second estimator 23. Specifically, thedetector 25 detects, as an obstacle, the three-dimensional points thatare not present on the movement plane out of the three-dimensional pointgroup obtained by the measurer 21.

FIG. 15 is an explanatory view illustrating an exemplary technique fordetecting the obstacle in the embodiment, and illustrates thethree-dimensional point group 101 obtained by the measurer 21 and themovement plane 104 estimated by the second estimator 23 on the yz plane.In the example illustrated in FIG. 15, the detector 25 calculates adistance d from the movement plane 104 in the y-axis direction for eachof the three-dimensional points included in the three-dimensional pointgroup 101, and detects the three-dimensional point having a distance dequal to or larger than an error as the three-dimensional point includedin the obstacle. The error is a measurement error in thethree-dimensional measurement performed by the measurer 21.

The output unit 27 performs output on the basis of the detection resultof the detector 25. For example, the output unit 27 causes a speaker(not illustrated) installed in the moving object 1 to output theposition of the detected obstacle as a voice, or a display installed inthe moving object 1 to display the position of the detected obstacle onan image acquired by the first acquirer 11.

FIG. 16 is a flowchart illustrating an exemplary flow of a procedure ofthe processing in the embodiment.

The first acquirer 11 sequentially acquires images captured in timeseries from the image capturing unit 5 while the second acquirer 13sequentially acquires the first position information and the firstdirection information (step S101).

The third acquirer 15 acquires the moving object information about othermoving object that moves in the surroundings of the moving object 1(step S103).

The identifier 17 identifies the moving object region in which the othermoving object is present for each of the images acquired by the firstacquirer 11 on the basis of the first position information and the firstdirection information that are acquired by the second acquirer 13 andthe moving object information acquired by the third acquirer 15 (stepS105).

The first estimator 18 extracts the feature points from the non-movingobject regions in the respective images acquired by the first acquirer11 in order to estimate the position and the posture of the imagecapturing unit 5. If the feature points sufficient for estimation areextracted (Yes at step S107), the first estimator 18 tracks (performtracking) the feature points among the images (step S109) and estimatesthe position and the posture of the image capturing unit 5 (step S111).

If the feature points sufficient for estimation are not extracted (No atstep S107) or if the estimation of the position and the posture of theimage capturing unit 5 fails (No at step S113), the processing ends.

If the estimation of the position and the posture of the image capturingunit 5 is successful (Yes at step S113), the searcher 19 searches for aplurality of sets of the corresponding points among the non-movingobject regions in the respective images acquired by the first acquirer11 (step S114).

The measurer 21 performs three-dimensional measurement on the basis ofthe position and the posture, which are estimated by the first estimator18, of the image capturing unit 5 and the multiple sets of thecorresponding points searched for by the searcher 19 to obtain thethree-dimensional points (step S115).

The second estimator 23 estimates the movement plane on which the movingobject 1 moves on the basis or the three-dimensional point groupobtained by the measurer 21 (step S117).

The detector 25 detects an obstacle on the basis of thethree-dimensional point group obtained by the measurer 21 and themovement plane estimated by the second estimator 23 (step S119).

The output unit 27 performs output on the basis of the detection resultof the detector 25 (step S121).

When the set of the corresponding points is searched for on the image onthe basis of a moving object 200 that is moving and serves as othermoving object as illustrated in FIG. 17, the position of a search point201 of the moving object 200 at previous image capturing time and theposition of a corresponding point 202 of the moving object 200 at thisimage capturing time differ from each other. If the three-dimensionalmeasurement is performed using the set of the search point 201 and thecorresponding point 201 as the set of the corresponding points, thethree-dimensional point (specifically, the depth) cannot be accuratelyobtained due to the principle of triangulation.

In contrast, when the set of the corresponding points are searched foron the image on the basis of a non-moving object 210 that does not movesuch as an object as illustrated in FIG. 18, a difference hardly occursbetween the position of a search point 211 on the non-moving object 210at previous image capturing time and the position of a correspondingpoint 211 on the non-moving object 210 at this image capturing time. Asa result, when the three-dimensional measurement is performed using theset of the search point 211 and the corresponding point 211 as the setof the corresponding points, the three-dimensional point (specifically,the depth) can be accurately obtained in the principle of triangulation.

The embodiment searches for the sets of the corresponding points among aplurality of images on the basis of the non-moving object regions otherthan the moving object regions in the respective images, thereby makingit possible to search for the sets of the corresponding points excludingthe moving object (other moving object). As a result, deterioration ofaccuracy of three-dimensional measurement can be reduced.

The embodiment can reduce the deterioration of accuracy ofthree-dimensional measurement as described above. When an obstacle isdetected using the three-dimensional points thus obtained, detectionaccuracy of the obstacle can be increased. For example, a tiny obstaclehaving a height about 10 cm can be accurately detected.

In the embodiment, as the change in the posture of the moving objectincreases, the threshold is reduced, thereby hardly deteriorating theaccuracy of three-dimensional measurement due to the influence of thechange in posture of the moving object even when the change increases.Furthermore, the movement plane of the moving object is estimated usingthe three-dimensional points near the moving object, thereby making itpossible to increase the accuracy of estimating the movement plane.

The embodiment can increase the accuracy of estimating the movementplane as described above. When an obstacle is detected using themovement plane thus estimated, the detection accuracy of the obstaclecan be more increased.

In the embodiment, the feature points are extracted from the non-movingobject regions when the position and the posture of the image capturingunit 5 are estimated. As a result, the positions of the feature pointsto be tracked hardly shift. Consequently, the deterioration of accuracyof estimating the position and the posture of the image capturing unit 5can also be reduced.

Modification

In the embodiment, the three-dimensional points each having a distance(distance in the height direction) equal to or larger than the errorfrom the movement plane are detected as the three-dimensional pointsincluded in the obstacle. The three-dimensional points each having adistance (distance in the height direction) equal to or larger than theerror from the movement plane and equal to or smaller than a fifththreshold may be detected as the three-dimensional points included inthe obstacle. In this case, when the fifth threshold is set to theheight (distance) nearly the same as the height of the moving object 1,obstacles such as a traffic signal and a pedestrian bridge are preventedfrom being mistakenly detected as the obstacles.

Hardware Structure

FIG. 19 is a schematic diagram illustrating an exemplary hardwarestructure of the measurement device of the embodiment and themodification. As illustrated in FIG. 19, the measurement device of theembodiment and the modification has a hardware structure utilizing anormal computer. Specifically, the measurement device includes a controldevice 901 such as a CPU, a main storage device 902 such as a read onlymemory (ROM) or a random access memory (RAM), an auxiliary storagedevice 903 such as a hard disk drive (HDD) or a solid state drive (SSD),a display device 904 such as a display, an input device 905 such as akeyboard or a mouse, and a communication device 906 such as acommunication interface.

A program executed by the measurement device in the embodiment and themodification is stored and provided in a computer-readable storagemedium, which may be provided as a computer program product, such as acompact disc read only memory (CD-ROM), a compact disc recordable(CD-R), a memory card, a digital versatile disc (DVD), or a flexibledisk (FD), as an installable or executable file.

The program executed by the measurement device in the embodiment and themodification may be stored in a computer connected to a network such asthe Internet and provided by being downloaded via the network.Furthermore, the program executed by the measurement device in theembodiment and the modification may be provided or distributed via anetwork such as the Internet. The program executed by measurement devicein the embodiment and the modification may be embedded and provided in aROM, for example.

The program executed by the measurement device in the embodiment and themodification has a module structure that achieves the respective unitsdescribed above in a computer. In practical hardware, the CPU reads outthe program from the ROM or the HDD to the RAM so as to execute theprogram, so that the respective units described above are achieved inthe computer.

The present invention is not directly limited to the embodiment and themodification. The invention can be embodied by changing componentswithout departing from the spirit and scope of the invention whenpracticed. In addition, various aspects of the invention can be made byproperly combining the components of the embodiment and themodification. For example, some components may be eliminated from all ofthe components of the embodiment and the modification. Furthermore, thecomponents of the different embodiments and modifications may beproperly combined.

For example, the steps in the flowchart of the embodiment may be changedin execution order, some steps may be executed simultaneously, or thesteps may be executed in different order every implementation withoutdeparting from their roles.

The embodiment and modification can prevent the deterioration ofaccuracy of three-dimensional measurement even when other moving objectis present in a plurality of images captured in time series.

While certain embodiments have been described, these embodiments havebeen presented by way of example only, and are not intended to limit thescope of the inventions. Indeed, the novel embodiments described hereinmay be embodied in a variety of other forms; furthermore, variousomissions, substitutions and changes in the form of the embodimentsdescribed herein may be made without departing from the spirit of theinventions. The accompanying claims and their equivalents are intendedto cover such forms or modifications as would fall within the scope andspirit of the inventions.

What is claimed is:
 1. A measurement device comprising: processingcircuitry configured to: acquire a plurality of images captured in timeseries by an image capturing unit installed in a moving object; acquirefirst position information that indicates a position of the movingobject and first direction information that indicates a direction of themoving object; acquire, through a communication device separated fromthe moving object by performing road-vehicle communication, movingobject information that includes second position information indicatinga position of other moving object moving in surroundings of the movingobject; identify a moving object region in which the other moving objectis present for each of the images, based on the first positioninformation, the first direction information, and the moving objectinformation; estimate a position and a posture of the image capturingunit based on the images; search for a plurality of sets ofcorresponding points among non-moving object regions other than themoving object regions in the respective images; and performthree-dimensional measurement based on the position and the posture ofthe image capturing unit and the sets of the corresponding points;estimate a movement plane on which the moving object moves based on athree-dimensional point group that is obtained as a result of performingthe three-dimensional measurement; and detect an obstacle based on thethree-dimensional point group and the movement plane, wherein thethree-dimensional point group is obtained by extractingthree-dimensional points each having a distance from the moving objectequal to or smaller than a threshold in a movement direction of themoving object, the threshold being set such that with an increase in atime series change in the posture of the image capturing unit, thethreshold is reduced.
 2. The device according to claim 1, wherein inestimating, the processing circuitry is configured to extract featurepoints from the respective images, tracks the feature points among theimages, and estimates the position and the posture of the imagecapturing unit.
 3. The device according to claim 2, wherein inestimating, the processing circuitry is configured to extract thefeature points from the non-moving object regions of the respectiveimages, tracks the feature points among the images, and estimates theposition and the posture of the image capturing unit.
 4. The deviceaccording to claim 1, wherein the non-moving object regions each have apredetermined size based on the position of the other moving object. 5.The device according to claim 1, wherein the non-moving object regionseach have a size according to a distance between the moving object andthe other moving object based on the position of the other movingobject.
 6. The device according to claim 1, wherein the moving objectinformation further includes model information that indicates a modelobtained by abstracting a shape of the other moving object, and thenon-moving object regions each correspond to a region calculated basedat least in part on the model of the other moving object and theposition of the other moving object.
 7. The device according to claim 6,wherein the moving object information further includes second directioninformation that indicates a direction of the other moving object, andthe non-moving object regions each further correspond to the regioncalculated based at least in part on the direction of the other movingobject and the position of the other moving object.
 8. The deviceaccording to claim 1, wherein the moving object information furtherincludes texture information that indicates at least one of a color anda pattern of the other moving object, and the non-moving object regionseach correspond to a region calculated based at least in part on thetexture information about the other moving object and the position ofthe other moving object.
 9. The device according to claim 1, wherein indetecting, the processing circuitry is configured to detect, as theobstacle, a three-dimensional point that is not present on the movementplane out of the three-dimensional point group.
 10. The device accordingto claim 1, wherein in acquiring the moving object information, theprocessing circuitry is configured to acquire the moving objectinformation from the other moving object.
 11. The device according toclaim 1, wherein in acquiring the moving object information, theprocessing circuitry is configured to acquire the moving objectinformation from a monitoring device that monitors the moving object andthe other moving object.
 12. A measurement method comprising: acquiringa plurality of images captured in time series by an image capturing unitinstalled in a moving object; acquiring first position information thatindicates a position of the moving object and first directioninformation that indicates a direction of the moving object; acquiring,through a communication device separated from the moving object byperforming road-vehicle communication, moving object information thatincludes second position information indicating a position of othermoving object moving in surroundings of the moving object; identifying amoving object region in which the other moving object is present foreach of the images based on the first position information, the firstdirection information, and the moving object information; estimating aposition and a posture of the image capturing unit based on the images;searching for a plurality of sets of corresponding points amongnon-moving object regions other than the moving object regions in therespective images; and performing three-dimensional measurement based onthe position and the posture of the image capturing unit and the sets ofthe corresponding points; estimating a movement plane on which themoving object moves based on a three-dimensional point group that isobtained as a result of performing the three-dimensional measurement;and detecting an obstacle based on the three-dimensional point group andthe movement plane, wherein the three-dimensional point group isobtained by extracting three-dimensional points each having a distancefrom the moving object equal to or smaller than a threshold in amovement direction of the moving object, the threshold being set suchthat with an increase in a time series change in the posture of theimage capturing unit, the threshold is reduced.
 13. A measurement devicecomprising: a processor; and a memory that stores processor-executableinstructions that, when executed by the processor, cause the processorto: acquire a plurality of images captured in time series by an imagecapturing unit installed in a moving object; acquire first positioninformation that indicates a position of the moving object and firstdirection information that indicates a direction of the moving object;acquire, through a communication device separated from the moving objectby performing road-vehicle communication, moving object information thatincludes second position information indicating a position of othermoving object moving in surroundings of the moving object; identify amoving object region in which the other moving object is present foreach of the images, based on the first position information, the firstdirection information, and the moving object information; estimate aposition and a posture of the image capturing unit based on the images;search for a plurality of sets of corresponding points among non-movingobject regions other than the moving object regions in the respectiveimages; and perform three-dimensional measurement based on the positionand the posture of the image capturing unit and the sets of thecorresponding points; estimate a movement plane on which the movingobject moves based on a three-dimensional point group that is obtainedas a result of performing the three-dimensional measurement; and detectan obstacle based on the three-dimensional point group and the movementplane, wherein the three-dimensional point group is obtained byextracting three-dimensional points each having a distance from themoving object equal to or smaller than a threshold in a movementdirection of the moving object, the threshold being set such that withan increase in a time series change in the posture of the imagecapturing unit, the threshold is reduced.